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A recent Letter [Q claims to have discovered evi- 
dence for birefringence in the propagation of radio waves 
across cosmological distances. Unfortunately, this claim 
is based on a flawed statistical analysis. 

To search for birefringence, the authors look for cor- 
relations between the direction and distance to a galaxy 
and the angle f3 between the polarization direction and 
the galaxy's major axis. Plotting the data as shown in 
Fig. 1(d) of their paper, here reproduced as Fig. 1, they 
use the correlation coefficient R^y as their statistic. 

To estimate the significance of their result, the authors 
use mock data samples constructed by randomly picking 
the angle /3 from a uniform distribution of allowed an- 
gles^ This is not the proper null hypothesis for testing 
the dependence of (3 on the direction and distance to 
the galaxies. Rather, for the null hypothesis, one should 
draw the angles from the observed distribution, which in 
the case of the high-redshift subsample, from which the 
primary conclusions were drawn, clearly is not a uniform 
distribution. From Fig. 1, one can see by eye that the po- 
larization in these galaxies tends to align with the galaxy 
minor axis, i.e. prefers ±7r/2 and avoids or ±7r. For 
example, ~ 3/4 of the points have 7r/4 < < 37r/4. 

That this matters can be easily seen from the following 
example. Consider the region in the x-y coordinate plane 
spanning —1 to 1 in both directions. If we uniformly fill 
the first and third quadrants, the correlation coefficient 
Rxy will be 0.75. If, however, we fix y = 0.5sgn(x) while 
allowing x to span —1 to 1 uniformly as before, then 
Rxy = ■\/3/2 « 0.867. Collapsing the y direction in this 
way allows more of the scatter to be explained by the 
best-fit line. 

Hence, we should expect that the tendency of the angle 
P to prefer ±7r/2 will cause Rxy to be higher than it would 
be if /3 were uniformly distributed between and ±7r. By 
using the latter as their null hypothesis, the authors find 
a spuriously high statistical significance for their result. 
Indeed, if the underlying galaxy population truly had a 
uniform intrinsic distribution of /3, it would be impossible 
to measure the proposed birefringence at all; one could 
not detect a rotation of such a distribution. 

Stated another way and estimating by eye, in Fig. |l| 
the data are more tightly correlated than they would be 




'^To be specific, they choose from random distributions the 
directions of each galaxy's major axis and plane of polariza- 
tion. This results in a uniform distribution (in the 1st and 
3rd quadrants) for the angle between these two directions. 
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FIG. 1. Figure 1(d) reproduced from [1]. 71 galaxies with 
redshifts above 0.3 are shown. 7 is the angle from the pro- 
posed birefringence direction, r is the distance to the galaxy, 
and P is the angle between the galactic major axis and the 
polarization direction. See [1] for more details. Our claim is 
that the l3 are not uniformly distributed between and ±7r 
but rather are clumped toward ±7r/2. 

if the (3 values were randomly and uniformly distributed 
between and ±7r. However, they are not significantly 
more correlated than they would be if the /3 values in a 
quadrant were shuffled among themselves while the best- 
fit line was adjusted accordingly. Hence, the claimed cor- 
relation of the angle f3 with the position and distance of 
the galaxy is not statistically significant. 

Taking the null hypothesis that the birefringence does 
not exist and that the angles between the polarization 
directions and galaxies' major axes are distributed as the 
data indicate, one is left to explain why the particular 
direction in the sky turned out to yield a higher R^y than 
other directions. This most likely results from combining 
the inhomogeneous sky coverage — the sample is mostly 
from the Northern sky and avoids low galactic latitudes — 
and the propensity of the chosen statistic Rxy to prefer 
directions that place many galaxies near the center of the 
spread in rcos7 where the tendency of /3 to prefer the 
center of its range can best reduce the scatter. It is not 
surprising that such a direction could exist. 

A second error relating to the choice of statistic and 
null hypothesis is the authors' use of the slope of the best 
fit line in Fig. |^ as a measure of the inverse birefringence 
scale Aj^. Because the null hypothesis (either the one 
they used or the one proposed here) produces a non-zero 
slope in the absence of birefringence, this is clearly a 
highly biased and inappropriate estimator. 
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Nodland & Ralston [1] claim to have discovered evi- 
dence for birefringence in the propagation of radio waves 
through the universe. We have argued [2] that flaws in 
their statistical analysis decrease the statistical signifi- 
cance of their results; Carroll & Field [3] reach similar 
conclusions. Further data from Leahy [4] and Wardle et 
al. [5] strongly argue against the proposed effect. In [6], 
Nodland & Ralston responded to our critique. Here, we 
explain why their response is incorrect. 

Nodland & Ralston base their reply [6] to our Com- 
ment on claims that several of our assertions were in 
error. They raise a number of different issues, but the 
chief arguments appear to fall into three categories. They 
claim 1) that the distribution of the angle (3 used in their 
Monte Carlo methods differs from the uniform distribu- 
tion we ascribed to it, and 2) that one could measure 
a birefringence even if the underlying angular distribu- 
tion(s) were uniform. They also claim 3) that our meth- 
ods would obscure a reasonable linear correlation even if 
it were in the data. We show here in detail that they are 
in error in these three claims. Indeed, we find nothing in 
their reply that gives us reason to modify our Comment. 

1) Nodland & Ralston object to our claim that the 
angle (3 in their simulations is drawn from a uniform dis- 
tribution. We show here that our assertion is correct, i.e. 
that the simulated (3's are indeed uniformly distributed. 

The angle used in their statistical analysis does indeed 
depend on the galaxy's position angle ip, its polarization 
angle Xj its position on the sky, and the proposed bire- 
fringence direction s. However, the analysis procedure 
splits into two pieces: 1) the determination of the angles 
(3~^ and [3^ from x ^-nd ip, and 2) the choice of which of 
these angles to use given the particular relative position 
of the galaxy and the birefringence direction (i.e. [3^ in 
one half-space and in the other). The search over 
many different s, as detailed in their procedures, never 
alters the values of (3'^ or (3~] it merely alters which one 
of them is used at any given time. Hence, once (3^ and 
(3~ are determined from x ^nd ij: at the beginning of the 
calculation, one need never refer to x ^-nd V' again. 

Moreover, the definitions of j3^ and (3~ reveal that (3^ 
= (3^ — TT. Hence, knowledge of /3+ for each galaxy at 
the beginning of the calculation, before calculating the 
correlation coefficient with respect to any birefringence 
direction, is all the information needed about the galaxy's 
polarization angles and position angles. This is true for 
both of the Monte Carlo procedures described in [1]. 

Hence, to prove that our treatment is equivalent to 
theirs, all we need do is show that drawing x a-nd "0 from 
independent uniform distributions yields a uniform dis- 
tribution of Z?"*". This is fairly clear, but for completeness, 
we now show the algebra. 

Assume that for a given galaxy the angles x a-nd ip are 
drawn from independent uniform distributions on [0,7r]. 
Now, consider the quantity A = x — "0- The probability 



density for this quantity is obtained by convolving the 
uniform distributions of x and ip. The result is a triangle- 
shaped distribution, 



/a(A)^^(^-|A|), 



(1) 



for all values of A between — tt and tt. (Here /a is an 
ordinary probability density: the probability that A lies 
between A and A -I- dA is /a (A) dA.) Now apply equa- 
tion (2) of [1] to convert A to P'^ . Given any /3+ between 
and TT, there are two values of A that can correspond 
to this /3+: either A — /3+ or A = /3+ — tt. Therefore, 
the probability density for /3+ is 



//3+(/3+) = /a(/3+) + /a(/3+-^) 



(2) 



= 4(^-/3+) + ^(^-(^-/3+)) = -. (3) 

So Z?"*" is uniformly distributed on [0,7r]. Hence, Nodland 
& Ralston's claim that our treatment differs from theirs 
is incorrect. 

Real radio galaxies do not have uniformly distributed 
values of p. This is clear from Figure Id of [1], in which 
1/3*1 is clearly seen to cluster around ±7r/2, and it is also 
well known from previous studies of radio galaxies. Our 
argument is that the incorrect assumption of uniformity 
artificially inflates the statistical significance of their re- 
sults. 

Nodland & Ralston defend their procedure by point- 
ing out that X and tp are uniformly distributed in the real 
data. That is correct but irrelevant. Only differences be- 
tween X and ip enter the calculation, and the correlation 
between the two angles causes these differences (encoded 
in (3^) to be distributed nonuniformly. By drawing x 
and ■0 from independent uniform distributions, Nodland 
& Ralston fail to take this correlation into account, lead- 
ing to incorrect results. 

2) Nodland & Ralston dispute our statement that "if 
the underlying galaxy population truly had a uniform 
distribution of /?, it would be impossible to measure the 
proposed birefringence at all." To see why our statement 
is true, remember that one is trying to measure an ad- 
ditional path-length-dependcnt variation on top of the 
intrinsic angle between the polarization and the major 
axis. If the polarization direction as the radiation left 
the galaxy were unrelated to the galaxy position angle 
i/), one would never be able to distinguish the initial po- 
larization angle from the rotation induced as the light 
traveled to us. Only by knowing something about the 
relation between the intrinsic polarization direction and 
some other observable property of the galaxy can one 
measure an additional path-length-dependent rotation. 

For a mathematical treatment of this, let us suppose 
for the moment that the direction s of the axis is fixed. 
Then as we have argued, the Nodland-Ralston null hy- 
pothesis is equivalent to drawing each value from 
a uniform distribution, generating (3~ , and then choos- 
ing the appropriate one according to the sign of cos 7. 
Now, suppose that we "rotate" each (3 in this data set 



by adding (modulo tt) the amount ^Aj^rcos7 to each 
p. Since the original /3's were independently and uni- 
formly distributed, the resulting /3's will have precisely 
the same statistical distribution as before: they will be 
independent and uniformly distributed. There is there- 
fore no statistical test that can distinguish between the 
"rotated" and "unrotated" data sets. 

Nodland & Ralston propose as a counterexample a 
data set in which there is a perfect linear relation be- 
tween (3 and rcos7. While this data set has a uniform 
distribution of the observed /3, it is not a case in which 
the underlying galaxy population satisfies this uniform 
distribution, and hence it has no bearing on the question 
at hand. Monte Carlo data sets drawn from uniform in- 
trinsic distributions of /3, with or without an additional 
^Aj^rcos7 rotation, would have a neghgible probabil- 
ity to place all of the points so nicely on a straight line. 
Since none of the Monte Carlo sets would look anything 
like the data (as quantified, say, by the correlation coef- 
ficient), this data would be not only striking evidence for 
birefringence but also evidence against an uniform distri- 
bution of intrinsic /3's. Indeed, it is exactly because the 
birefringent model would reduce the /3 distribution to 
a distance-independent distribution that one favors this 
interpretation. Hence the counterexample does not bear 
upon our claim, as it does not satisfy the supposition of 
our assertion. 
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3) Nodland & Ralston claim that shuffling the data 
would fail to detect a perfectly correlated S{y — x) dis- 
tribution. This is incorrect. Shuffling means randomly 
matching the x coordinate of one data point with the 
y coordinate of another. If the data lay on an inclined 
straight line, no permutations of the data would ever pro- 
duce a data set with as high a correlation; the new data 
sets would generically show large amounts of scatter. In 
this particular case, none of the shuffled data sets would 
have as high a correlation as the original data, since the 
latter has the maximum possible correlation coefficient 
(r = 1). Hence, the correlation would be detected at 
high significance. 

In the latter half of their reply, Nodland & Ralston 
merely restate our case. When they performed a some- 
what more correct statistical procedure (shuffling the 
/3's), the statistical significance of their result dropped 
considerably. They claim the signal is still significant, 
but they do not account for the fact that one has looked 
at many possible directions in the sky. In short, they 
confirm our essential point that using the distribution of 
/3's from the observed distribution increases the corre- 
lation measured in the non-refringent universe, thereby 
decreasing the statistical significance of their claims. A 
similar calculation may be found in Carroll & Field [3, 
p. 10]. 

In conclusion, we find that none of the arguments in 
[6] provide us with any reason to modify the conclusions 
of our original Comment. 



